Machine unlearning focuses on the computationally efficient removal of specific training data from trained models, ensuring that the influence of forgotten data is effectively eliminated without the need for full retraining. Despite advances in low-dimensional settings, where the number of parameters \( p \) is much smaller than the sample size \( n \), extending similar theoretical guarantees to high-dimensional regimes remains challenging. We propose an unlearning algorithm that starts from the original model parameters and performs a theory-guided sequence of Newton steps \( T \in \{ 1,2\}\). After this update, carefully scaled isotropic Laplacian noise is added to the estimate to ensure that any (potential) residual influence of forget data is completely removed. We show that when both \( n, p \to \infty \) with a fixed ratio \( n/p \), significant theoretical and computational obstacles arise due to the interplay between the complexity of the model and the finite signal-to-noise ratio. Finally, we show that, unlike in low-dimensional settings, a single Newton step is insufficient for effective unlearning in high-dimensional problems -- however, two steps are enough to achieve the desired certifiebility. We provide numerical experiments to support the certifiability and accuracy claims of this approach.

Due to legal requirements like the European General Data Protection Regulation (GDPR), the California Consumer Privacy Act, and many others, users gain more control over their personal data collected daily. The right to be forgotten is of particular importance, which states that collected data must be deleted when requested. Deleting data is often insufficient to provide real data privacy. This is especially the case if the data was used to train machine learning models since they might expose information about their training data via white-box or even black-box access. Motivated by this, the field of Machine Unlearning and Forgetting gained more and more attention.